Converting Real World Objects to 3D Models

Converting Real-World Objects to 3D Models

When developing an application in VR, it might be necessary to create 3D models mirroring objects from the real world. If you have little experience in 3D modeling or have to generate rather complex shapes, this task might be tedious.

Luckily, smartphones and image processing techniques have evolved far enough to help with this problem. Using your smartphone, you can scan an object using images and create models automatically.

Apps for Scanning

Scanning can be done with the following apps:

• (Android/iOS) Polycam - 3D Scanner: The base version is free but only allows scanning a few objects and allowing exports only to gITF. The Pro version is a subscription costing €8.99 per month with unlimited scans and all export types.
  The restriction to the gITF format is okay, since it is readable and convertible by blender (See Cleaning Up the Scans for details).
  The major downside is the restriction of scans, as they often have to be redone if the model was not scanned correctly. As for small and medium objects, it produces good scans.
• (Android) Qlone 3D Scanner: While costing €21.99, it provides unlimited scans. It requires a printed paper guide for tracking and is great for scanning small objects, that are not flat or shiny.
• (Android/iOS) MagiScan: Produces nice results and is easy to use.
  The pricing seems fair: Currently €2.69 for 10 Scans and €42.99 for 250 scans. However the amount of free scans is only 3, which might be not enough for proper testing.
• (iOS) Trnio 3D Scanner: Costing only €4.99 this might be a good option when using an iOS device.
• (PC) 3DF Zephyr Free: The biggest benefit is that it is free, but with the downside that the images must be transmitted to the PC first. The benefit of doing that is that the other apps require cloud computing for mesh creation. Using Zephyr, this can be done offline, which is useful if sensible data should be scanned. A great German tutorial can be found here.
  The greatest limitation is that a maximum of 50 images are allowed. This might seem like a lot, but it is recommended to use way more, especially with larger objects. The restriction can be lifted by using the Lite or Full version, which comes at a price point of €149(+VAT) or €3,900(+VAT) respectively.

Tips for Scanning

When scanning, the quality of the scan is greatly dependent on the amount and quality of the images.

To achieve the best results, try to do the following:

• Use a clean, non-complex underground and background if possible (e.g., white table in a white room).
• Try to light up your room properly (not too dark, no hard light). If the lighting of your application is not too bad (e.g., very dark), also try to match that.
• Allow yourself to travel around the object or put the object on a pedestal that can be rotated easily.
• If the underside should also be scanned, try to either hang it or impale the object with toothpicks.
• Make as many pictures as possible from every angle of the object, as well as its features. This may range from 30 to multiple hundreds, depending on the size and complexity of your object and the restrictions of your app.
• If your models might not look to good, make more (with the exact same setup), do a completely new scan, or check your images to try to remove bad ones (e.g. blurry, ...)
• Usually being too close to the object gives worse results, as there are less hints fewer hints as to where the picture belongs to.

The there are many 3d scanning service providers, like the Digital Humanity Center in Tübingen or other institutions near you. They use special equipment resulting in way better and quicker results, than with a smartphone. If you are having any issues, feel free to reach out to them.

Cleaning Up the Scans

When doing your first scan, you will notice that the scan contains much more than just the model. Upon further inspection, you will see that the model's mesh is way more complex (uneven) than it should be, and its textures seem to be stitched from hundreds of little patches. Unoptimized models will have a big negative performance impact in your application, thus require cleanup and simplification. Editing the scans will also allow you to change to another format that is accepted by Unity.

All of this can be easily achieved by using Blender. Note that the following description is based on Blender and its features and shortcuts. The workflow may vary with other programs.

1. Import the model into Blender. Go to "File > Import > ..." and select the file extension of your model.
2. Rotate the object to face to the forwards direction (in Unity). The front should be facing towards positive y.
3. Scale the Object to the correct size (if necessary).
4. Move the object to the origin and set its Origin by pressing "RMB" and selecting "Set Origin > Origin to 3D Cursor".
5. Clean up any floating geometry and remove the outlines roughly. Select the object and press "Tab" to switch to Edit Mode. Press "1" to select Vertices, then select vertices that are not part of the object using any of the selection tools. Then delete the selection using the "Delete"-Key (German: "Entf") and select vertices. Pressing "Alt + Z" will turn on x-ray mode, allowing you to select all vertices in the selection, not only those visible. Be careful not to delete too much from the scanned object itself, even if this might result in some jagged edges. It is okay to remove parts of the object that are messed up and should be removed, e.g., sticking out.
6. To square up the edges, the "Boolean"-Modifier will be used. Go to "Object Mode" (Press "Tab") and the spawn in a shape using "Shift + A" and select "Mesh > ...". When having a scan that has one flat base, either a cube or cylinder should work. Scale and move the shape to fit over the Model. It should be intersecting in such a way that the parts that should be kept are contained inside the cube/sphere. Then select the Shape and Go to the "Modifier"-Tab in the dialog on the right (the little wrench). Select "Add Modifier > Boolean" and select Intersection as Boolean-Operation. Use the eyedropper (German: "Pipette") to select the original mesh of your scan. Then apply the modifier using "Ctrl + A" or selecting the option using the arrow pointing downward in the topmost line of the boolean-modifier-window. After applying the modifier, it will vanish, creating a new mesh with clean edges. For the next steps, this mesh will be used, instead of the original. Its textures will look wonky, but this will be fixed later! If you do not have any textures, you chose the wrong mesh to apply the boolean modifier
7. If there are any holes in the model, you will need to close them now. Select the new model and switch to vertex editing as described in step 4. Using edge loops to select the holes may not work, as the models' geometry is not that good. Instead, select all vertices along the edge of a hole and press "Alt + F" to fill the hole. This might not work if the hole is too big/complex or if the hole is on the edge of the model, e.g., if there is not a full loop of edges surrounding the hole. To fix that, select two vertices (Shift + LMB) that enclose a section of the hole nicely or that would form the edge of the model. Press "F" to insert an edge to connect them. If the edge is particularly long, press "2" to go to edge selection and select that edge. Pressing "RMB" and selecting subdivide will insert another vertex in the middle of that edge. Then proceed to close the holes with "Alt + F" or, optionally, but not recommended, with "F".
8. When all holes are closed, it is time to simplify the geometry. This is very important, as it will speed up remeshing and improve performance in Unity. The Object may look a bit blocky, but this will be fixed using a Normal-Maping of the Original Mesh. For the reducing the poly-count will have a number of different options depending on your Model-Type:
   • Decimating (Simple): The simplest form of reducing the poly count is "Decimation" but it will leave a number of bad geometry. Switch to Object Mode and add a "Decimate"-Modifier (like the Boolean one in step 5). Decrease the Ratio as much as possible without loosing too much detail, then apply the Modifier. A value of 0.2 is a good starting point.
   • Remeshing via Modifier (for nearly closed models): Add the "Remesh"-Modifier, select the "Sharp" option (or "Smooth" depending on the application), and increase the "Octree Depth" and "Scale" until a satisfiable model can be seen. Be careful: Large values of "Octree Depth" may cause serious performance drops. Applying the Modifier will also solidify the Object.
   • Remeshing via Quads (for rooms): Select the Object in Object-Mode and select the Object Data Modifier-Tab (Green Triangle) from the Property-Tab on the bottom right. Unfold the Option for Remesh and select the "Quad"-Option and press the button. Choose a polygon count you think is right and hit "Ok". If the Model has too few faces and looks too blocky, undo the step (Ctrl + Z) and increase the "Number of Faces".
9. (Optionally) Apply the "Smooth"-Modifier to get a smoother model.
10. (Optionally) For further sculpting, it might be necessary to increase the polygon count again by subdividing it via the corresponding Modifier. Then Change the Tab at the Top-Center to "Sculpting" to sculpt. You find some further tutorials online. Be sure to reduce the poly count later again.
11. Once the mesh is done, it is time to add back the lost details and fix its texture. To do this, follow this tutorial. This should leave you with a normal map and base-color texture on your low-poly mesh.
12. If some parts of the texture map were missing or looking wrong, go to the "Texture Painting"-Tab and alter the texture there.
13. Finally, select only the low-poly mesh and select "File > Export..." and Export to .obj or .fbx. Be sure to limit to "Selected Object" by checking the box on the right of the export window.

Some helpful videos for similar workflows can be found here:

1. Capturing and Rough Cleanup
2. Further Cleanup
3. Reduce Poly and Bake Textures

Using Scans for Manual Remodeling (of Rooms)

It might be the case that the scans came out rather badly or that a very clean, detailed room or model is needed. This process will take time and requires some level of experience, but it comes with the benefit of reducing the complexity of the model immensely. This is especially beneficial for the performance of the application, as fewer faces require less computation.

Here are some steps for it:

1. Start with the layout. If you have a model, import it as described above and clean it up if necessary (No need for remeshing and textures). If you have a 2D-blueprint of the floor, also import it as an image and scale it to size.

2. Then add a plane and align it with one side of the room. Add more vertices in top-down view by selecting one vertex and "Ctrl+RightClick"-ing at the corners of the room, using the blueprint/scan as guide. Close the "loop" of vertices by selecting the last pair of vertices that shall be connected and press "F".

3. Move all Vertices of the Plane by pressing "A", then "G", "Z" and finally "0".

4. Fill the rest of the plane by selecting the vertices that surround the area to be filled and press "Alt+F".

5. Create the Walls by selecting all "wall edges" and press "Shift+E" to extrude. Then left-click at the correct height, using the model as a guide.

6. Press "Alt+F" again to form the ceiling.

7. Windows, Indentations and other features can be created by creating a new Shape ("Shift+A" in Object-Mode) and applying a Boolean-Modifier to the Room using this helper object.

8. For texturing certain faces (e.g. walls, floors, or windows) uniformly, they can be extracted into a separate object. To do this, select the desired faces and press "P" then choose "Selection" from the popup.

9. Add textures either by:

   • Baking them from the Model (See here)
   • Downloading some materials from the internet and importing them into blender
   • Using photographs from the actual location
   • Using the image data from the scan itself

   Create a material and image texture like this and add all textures you have to it (e.g., by using Gimp or Photoshop). If there is only one, simply use that as Texture for the Blender-Material. Then go to "UV Editing" and perform a "Smart UV Project" (also described in the video). Then move the faces to match their location and size on the texture.

More general tips:

• Creating simple Arches in Blender can be done by using the Bevel-Tool or Merging Edges/Faces together.
• When using a uniform/repeating texture for a material, try making it seamless for better repetition. This can either be done online or using GIMP under "Filters > Map > Make Seamless" (German: "Filter > Abbilden > Nahtlos kacheln...")
• Photographs usually suffer from perspective distortion. This can be removed by using the "Perspective"-Tool (3D-Wire-Cube Icon in Gimp).