Home   Donators   Contact Us       

<< Previous Tutorial     Next Tutorial >>

Tutorial 10 - Take your pick


3D scenes are represented in computer memory by making use virtual 3D models with vertices and polygons. These 3D models are then rendered to a 2D output device such as a computer screen. So how do we pick objects in a 3D scene using a 2D input device such as a mouse?

We can achieve 3D object picking by making use of colors. The image below shows two cube models (one cube in front of the other), with a (x,y) mouse click that occurred roughly at the center of the image. So how can we determine which model the user intended to select?

Firstly, we will assume that when the user makes a (x,y) selection in the 3D scene, that the object closest to him/her is the model that needs to be picked. Let's render the scene again, but this time we use an unique single color for each model in its entirety, with the OpenGL lighting feature disabled. The result might look something similar to the following image:

All we need to do now is to read the rendered OpenGL buffer and get the color of the pixel at the (x,y) coordinate of the mouse. In the example above we'll get an orange color. We can then determine from the orange color, that Model B was picked by the user.

Although this picking technique requires two renders of our scene, once for display and once for picking, it provides a very easy and effective way to give users the ability to pick 3D models using a 2D input device.

Tutorial Steps
Tutorial created with Real Studio 2011 Release 4.3.
1. Open Real Studio.
2. Choose the "Desktop" project template.
3. Add an OpenGLSurface control to Window1.
4. Resize and position OpenGLSurface1 to fill the whole form.
5. Tick the LockRight and LockBottom properties of OpenGLSurface1.
6. Save your project.
7. If you completed Tutorial 9 - Order in chaos, then open the project file with Real Studio and export the following classes and modules (save them next to your new project file):
  • R3DTColor
  • R3DTVector
  • R3DTPolygon
  • R3DTModel
  • R3DT_Models
  • R3DT_OpenGL
If you have not completed the previous tutorial, download the files from the links below (save them next to your new project file):
8. Import the classes and modules into your new project file. (Select File > Import... from the main menu)
9. Add a property named "MapName" of type Integer to the R3DTModel class.
10. Add a second method named "SetPosition" to the R3DTModel class with a parameter list of "newPos As R3DTVector".
11. Add the following code to SetPosition(R3DTVector):
' assign the new value to the position object

Position.x = newPos.x
Position.y = newPos.y
Position.z = newPos.z
12. Add a second method named "SetColor" to the R3DTModel class with a parameter list of "newCol As R3DTColor".
13. Add the following code to SetColor(R3DTColor):
Dim i as Integer ' loop variable
Dim poly As R3DTPolygon ' temporary polygon object

' loop through all the polygons and update their colors with the new value

for i = 0 to Polygon.Ubound

  poly = Polygon(i) ' get next polygon

  ' assign the color values

  poly.SurfaceColor.Red = newCol.Red
  poly.SurfaceColor.Green = newCol.Green
  poly.SurfaceColor.Blue = newCol.Blue

next i
14. Add a class named "R3DTScene" to the project.
15. Add a property array named "Model()" of type R3DTModel to the R3DTScene class.
16. Add a method named "R3DT_InitOpenGL" to the R3DT_OpenGL module.
17. Add the following code to R3DT_InitOpenGL:
' enable depth buffer to ensure objects in distance is not drawn over objects closer to us

OpenGL.glEnable OpenGL.GL_DEPTH_TEST
OpenGL.glDepthMask OpenGL.GL_TRUE

' make sure only back faces are culled and enable culling

OpenGL.glCullFace OpenGL.GL_BACK
OpenGL.glEnable OpenGL.GL_CULL_FACE

' enable lighing

OpenGL.glEnable OpenGL.GL_LIGHTING
18. Add a method named "R3DT_IntegerToColor" to the R3DT_OpenGL module with a parameter list of "intVal As Integer".
19. Set the return type of R3DT_IntegerToColor to R3DTColor.
20. Add the following code to R3DT_IntegerToColor:
Dim colMB As new MemoryBlock(4) ' memory block used to store color value
Dim col As R3DTColor

' load the integer map name value into the memory block

colMB.UInt32Value(0) = intVal 

' use the red, green, and blue values in the memory block to instantiate a color object

col = new R3DTColor(colMB.UInt8Value(0), colMB.UInt8Value(1), colMB.UInt8Value(2))

return col ' return the color to the caller
21. Open the R3DT_RenderModel method in the R3DT_OpenGL module.
22. Change the parameter list of R3DT_RenderModel to "model As R3DTModel, Optional PickModel As Boolean = False".
23. Remove all the code from R3DT_RenderModel and replace it with the following new code:
Dim i As Integer // loop variable
Dim j As Integer // loop variable
Dim poly As R3DTPolygon ' temporay polygon used for drawing
Dim vertex As R3DTVector ' temporary vertex used for drawing
Dim col As R3DTColor ' temporary object to store color

OpenGL.glPushMatrix ' save the current matrix

' set the position of the model

OpenGL.glTranslatef model.Position.x, model.Position.y, model.Position.z

' if picking model, set the color once using the model's map name 
' all the models of this polygon will be drawn with the same color when drawn for picking
if PickModel then
  col = R3DT_IntegerToColor(model.MapName) ' convert the map name to a color
  OpenGL.glColor3ub(col.Red, col.Green, col.Blue) ' set the color of the model
end if

' loop through all the polygons of the model and draw each one

for i = 0 to model.Polygon.Ubound

  ' get the next polygon from our model

  poly = model.Polygon(i)

  ' set the color of the polygon only if we are not picking models

  if not PickModel then
    OpenGL.glColor3d poly.SurfaceColor.Red, poly.SurfaceColor.Green, poly.SurfaceColor.Blue
  end if

  ' set the normal of the polygon

  OpenGL.glNormal3d poly.Normal.x, poly.Normal.y, poly.Normal.z

  ' draw the polygon

  OpenGL.glBegin OpenGL.GL_POLYGON ' start drawing of polygon

  ' loop through all the vertices of the current polgon

  for j = 0 to poly.VertexIndex.Ubound
    ' get the next vertex of the polygon
    vertex = model.Vertex(poly.VertexIndex(j))
    ' add the vertex to the OpenGL vertex list for the current polygon
    OpenGL.glVertex3d vertex.x, vertex.y, vertex.z
  next j

  OpenGL.glEnd ' end drawing of polygon

next i

OpenGL.glPopMatrix ' restore the matrix we saved before rendering the model
24. Add a method named "R3DT_RenderScene" to the R3DT_OpenGL module with a parameter list of "scene As R3DTScene, Optional PickModel As Boolean = False".
25. Add the following code to R3DT_RenderScene:
Dim i As Integer ' loop variable
Dim model As R3DTModel ' tempory model used for rendering

OpenGL.glPushMatrix ' save matrix

OpenGL.glLoadIdentity() ' load identity matrix

' lighting needs to be disabled when drawing a model map for picking

if PickModel then
  OpenGL.glDisable OpenGL.GL_LIGHTING ' disable lighting
end if

' clear the background and depth buffer

OpenGL.glClearColor(0, 0, 0, 1)

' move back a bit so that we can see the object

OpenGL.glTranslatef 0.0, 0.0, -20.0

' loop through all the models in the scene

for i = 0 to Scene.Model.Ubound

  ' get the next model

  model = Scene.Model(i)

  ' render the model

  R3DT_RenderModel model, PickModel

next i

' enable lighting again if we disabled it for model picking

if PickModel then
  OpenGL.glEnable OpenGL.GL_LIGHTING ' enable lighting
end if

OpenGL.glPopMatrix ' restore matrix
26. Add a method named "R3DT_PickModel" to the R3DT_OpenGL module with a parameter list of "scene As R3DTScene, x As Integer, y As Integer".
27. Set the return type of R3DT_PickModel to R3DTModel.
28. Add the following code to R3DT_PickModel:
Dim model As R3DTModel ' this variable will hold the model that is picked
Dim i As Integer ' loop variable
Dim pixelData As New MemoryBlock(4) ' holds pixel data read from OpenGL buffer
Dim pickMapName As Integer

' first we need to set up  map names for all the models in the scene

for i = 0 to scene.Model.Ubound
  scene.Model(i).MapName = i + 1
next i

' clear the background and depth buffer

OpenGL.glClearColor(0, 0, 0, 1)

' render the model map of the scene

R3DT_RenderScene scene, True

' get the pixel value at the given (x, y) position

OpenGL.glReadPixels(x, y, 1, 1, OpenGL.GL_RGB, OpenGL.GL_UNSIGNED_BYTE, pixelData)

' store the color of the pixel as the map name

pickMapName = pixelData.UInt32Value(0)

' did we get a pixel at (x, y) other than black?
if pickMapName > 0 then

  ' yes, so get the model with this mapname

  i = 0
  while (i <= scene.Model.Ubound) and (model = nil)
    ' did we find the model with the picked map name?
    if scene.Model(i).MapName = pickMapName then
      ' yes, so assign it to our return model object
      model = scene.Model(i) 
    end if
  i = i + 1 ' move to next model

end if

return model ' return the model object to the caller
29. Add a property named "Scene" of type R3DTScene to Window1.
30. Add the following code to the Open event of Window1:
Dim i As Integer ' loop variable
Dim model As R3DTModel ' tempory model object
Dim pos As new R3DTVector(0, 0, 0) ' vector to store random position
Dim col As new R3DTColor(1, 1, 1) ' object to store random color
Dim rnd as New Random ' object to generate random numbers with

' instantiate the Scene object

Scene = new R3DTScene

' generate random models in our 3D scene

for i = 1 to 100

  ' generate random position

  pos.x = rnd.InRange(0, 30) - 15
  pos.y = rnd.InRange(0, 30) - 15
  pos.z = rnd.InRange(0, 10) - 5

  ' generate random color

  col.Red = rnd.InRange(1, 100) / 255
  col.Green = rnd.InRange(100, 255) / 255
  col.Blue = rnd.InRange(1, 255) / 255

 ' instantiate a new cube model

  model = R3DT_GetModel_Cube

  model.SetPosition pos ' set the position of the cube
  model.SetColor col ' set teh color of the cube
  Scene.Model.Append model ' add the model to our scene

next i
31. Add the following code to the Paint event of Window1:
' refresh the OpenGL surface

32. Add the following code to the Open event of OpenGLSurface1:
R3DT_InitOpenGL ' initalize OpenGL environment

' instantiate memory blocks used to configure our light with

Dim light_position As new MemoryBlock(16)
Dim light_ambience As new MemoryBlock(16)
Dim light_diffANDspec As new MemoryBlock(16)

' define position of the light

light_position.SingleValue(0) = 0.0
light_position.SingleValue(4) = 0.0
light_position.SingleValue(8) = 1.0
light_position.SingleValue(12) = 0.0

' define ambience of the light

light_ambience.SingleValue(0) = 0.0
light_ambience.SingleValue(4) = 0.0
light_ambience.SingleValue(8) = 0.0
light_ambience.SingleValue(12) = 1.0

' define diffuse and specular of the light

light_diffANDspec.SingleValue(0) = 1.0
light_diffANDspec.SingleValue(4) = 1.0
light_diffANDspec.SingleValue(8) = 1.0
light_diffANDspec.SingleValue(12) = 1.0

' apply light settings

OpenGL.glLightfv OpenGL.GL_LIGHT0, OpenGL.GL_POSITION, light_position ' set position
OpenGL.glLightfv OpenGL.GL_LIGHT0, OpenGL.GL_AMBIENT, light_ambience ' set ambience
OpenGL.glLightfv OpenGL.GL_LIGHT0, OpenGL.GL_DIFFUSE, light_diffANDspec ' set diffuse
OpenGL.glLightfv OpenGL.GL_LIGHT0, OpenGL.GL_SPECULAR, light_diffANDspec ' set specular

OpenGL.glEnable OpenGL.GL_LIGHT0 ' enable our light source

' enable overwriting of material properties with vertex colors

33. Add the following code to the Resized event of OpenGLSurface1:
' set the viewport rectangle

OpenGL.glViewport 0, 0, OpenGLSurface1.Width, OpenGLSurface1.Height

' set up the perspective projection settings

OpenGL.glMatrixMode OpenGL.GL_PROJECTION
OpenGL.gluPerspective 60.0, OpenGLSurface1.Width / OpenGLSurface1.Height, 1, 100.0

' select and reset the modelview matrix

OpenGL.glMatrixMode OpenGL.GL_MODELVIEW
34. Add the following code to the Render event of OpenGLSurface1:
' render the Scene with OpenGL

R3DT_RenderScene Scene
35. Add the following code to the MouseDown event of OpenGLSurface1:
Dim model As R3DTModel ' model that was picked
Dim rnd As Random ' object to generate random numbers with

' pick the model located at the (x, y) position of the mouse
' the OpenGL y axis increase from bottom to top, so we need to ajust our y value to accomondate this


model = R3DT_PickModel(Scene, x, OpenGLSurface1.Height - y)

' was there a model located at this position?
if model <> nil then

  rnd = new Random

  ' yes, so lets give the model a new random color

  model.SetColor (rnd.InRange(100, 255) / 255, rnd.InRange(1, 200) / 255, rnd.InRange(1, 200) / 255)

  OpenGLSurface1.Render ' refresh the OpenGL view

end if
36. Save and run the project.
37. Click on the different cubes and see what happens.

Code Analysis

Let's first have a look at the new methods and properties added to the classes we created in Tutorial 9 - Order in chaos.

The MapName property that we added to the R3DTModel class will be used to store the Integer representation of the "RGB-map-color" used to identify the model during our picking routine.

We added a second SetPosition method to the R3DTModel class that takes a single R3DTVector parameter, instead of separate x, y, and z values. Strictly speaking this method does not expand the capabilities of our R3DTModel class, but simply makes it easier for us to set new model positions using existing vector objects. The addition of a second SetPosition method with a different parameter signature, is better known as method overloading or polymorphism in the world of computer science.

Similarly, we added a second SetColor method to R3DTModel, that simply makes things easier when working with models and existing R3DTColor color objects.

A new class, named R3DTScene is added to our project. This class has an array (Model()) that holds all stores all the model objects of your scene. The image below provides a basic illustration of the purpose of the R3DTScene class.

With the R3DT_InitOpenGL method added to the R3DT_OpenGL module, we started to create a initialization function for your 3D engine. In future tutorials and projects we can now simply call the R3DT_InitOpenGL method in the OpenGLSurface.Open event, instead of retyping the same code each time.

The new R3DT_IntegerToColor method is a helper function we use in our picking routines. It simply converts an Integer to a corresponding RGB color value.

A new optional parameter named PickModel (with its default value set to False), is added to our R3DT_RenderModel method. Instead of writing a new rendering routine to pick models, we tweak the existing render routine so that we can use it for both display and pick rendering. Using one method, for both display and picking, makes maintainability easier in the future when we improve and optimize the rendering routine.

The R3DT_RenderScene is similar to the R3DT_RenderModel method, with the difference being that it renders all the models found in a scene to the screen, instead of just a single model. Take note that the first parameter passed to it is of the type R3DTScene. It also has a PickModel argument (with its default value set to False), so that we can use it for both display and pick rendering.

R3DT_PickModel is the method that does the magic of picking a model in a 3D scene using 2-dimensional x and y values. Note that we only pass it a R3DTScene object, together with the required x and y values, and it will automatically return the R3DTModel object that was picked. If the given (x,y) coordinate does not contain a model (e.g. a black area on the screen), then R3DT_PickModel will return a Nil value.

Finally we put our R3DT_PickModel to good use on line 7 in the OpenGLSurface1.MouseDown event. There is one last concept we need to understand though. Notice how we give the value (OpenGLSurface1.Height-y) instead of y, when calling R3DT_PickModel. The coordinate system of OpenGL is implemented a little bit different than the coordinate system used for computer screens. A computer screen's y-axis increases from top to bottom, whereas the y-axis in OpenGL increases from bottom to top (see image below). We therefore need to match the screen (x,y) coordinate to the OpenGL (x,y) coordinate, by subtracting y from the total height of the OpenGL surface.

That concludes this tutorial. See if you can replace the code in the Window1.Open with your own code, and build your own scene made up from 3D cube models.

Project Downloads

<< Previous Tutorial     Next Tutorial >>

All the content on Real 3D Tutorials, with the exception of the SyntaxHighlighter which is licensed under the MIT License, is provided to the public domain and everyone is free to use, modify, republish, sell or give away this work without prior consent from anybody. Content is provided without warranty of any kind. Under no circumstances shall the author(s) or contributor(s) be liable for damages resulting directly or indirectly from the use or non-use of the content.
Should you find the content useful and would like to make a contribution, you can show your support by making a donation.