Voxel Carving of 3D Surfaces using AR Markers for Pose Estimation

Final project for "3D Scanning and Motion Capture" class in SS2022 at Technical University of Munich.

Table of Contents

• 1. Abstract
• 2. Setup
  • 2.1. Clone the repository
  • 2.2. Windows
    • 2.2.1. CMake
    • 2.2.2. Git BASH
    • 2.2.3. Eigen
    • 2.2.4. OpenCV
  • 2.3. Other OS
  • 2.4. Build the Project
• 3. Example
• 4. How to run?
  • 4.1. Generate ChArUco Board
  • 4.2. Camera Calibration
  • 4.3. Segmentation
  • 4.4. Carve your own model
  • 4.5. Benchmarking
• References

1. Abstract

TODO

2. Setup

Install Eigen (Version 3.4).

Install OpenCV Library (Visual Studio 2019 VC16).

2.1. Clone the repository

Repository Structure

• Data

  • demo_material

    • images

    • masks

• Libs

• src

• CMakeLists.txt

• README.adoc

Data

You can use Data to store images, masks, and calibration files of models you want to carve. The subfolder demo_material contains all inputs and outputs used to generate the model as described in section 4.

Libs

If you have never worked with OpenCV or Eigen you can follow the instructions below and install these libraries in the Libs directory.

src

This directory contains the implementation of this project.

2.2. Windows

2.2.1. CMake

1. Download CMake (3.9.1 or newer)

2. Install CMake in your preferred directory.

3. Add CMake to PATH variable.

2.2.2. Git BASH

1. Download and install Git Bash (2.14.1 or newer)

2.2.3. Eigen

Download and extract Eigen (see Eigen - Getting started).

2.2.4. OpenCV

1. Navigate to the directory where you wish to install OpenCV (eg. {directory}/AR_Voxel_Project/Libs)

2. create new file installOCV.sh with the following content:

   #!/bin/bash -e
   
   # taken from: https://docs.opencv.org/3.4/d3/d52/tutorial_windows_install.html
   
   myRepo=$(pwd)
   CMAKE_GENERATOR_OPTIONS=-G"Visual Studio 17 2022"
   #CMAKE_GENERATOR_OPTIONS=-G"Visual Studio 16 2019"
   #CMAKE_GENERATOR_OPTIONS=-G"Visual Studio 15 2017 Win64"
   #CMAKE_GENERATOR_OPTIONS=(-G"Visual Studio 16 2019" -A x64)  # CMake 3.14+ is required
   if [  ! -d "$myRepo/opencv"  ]; then
       echo "cloning opencv"
       git clone https://github.com/opencv/opencv.git
   else
       cd opencv
       git pull --rebase
       cd ..
   fi
   if [  ! -d "$myRepo/opencv_contrib"  ]; then
       echo "cloning opencv_contrib"
       git clone https://github.com/opencv/opencv_contrib.git
   else
       cd opencv_contrib
       git pull --rebase
       cd ..
   fi
   RepoSource=opencv
   mkdir -p build_opencv
   pushd build_opencv
   CMAKE_OPTIONS=(-DBUILD_PERF_TESTS:BOOL=OFF -DBUILD_TESTS:BOOL=OFF -DBUILD_DOCS:BOOL=OFF  -DWITH_CUDA:BOOL=OFF -DBUILD_EXAMPLES:BOOL=OFF -DINSTALL_CREATE_DISTRIB=ON)
   set -x
   cmake "${CMAKE_GENERATOR_OPTIONS[@]}" "${CMAKE_OPTIONS[@]}" -DOPENCV_EXTRA_MODULES_PATH="$myRepo"/opencv_contrib/modules -DCMAKE_INSTALL_PREFIX="$myRepo/install/$RepoSource" "$myRepo/$RepoSource"
   echo "************************* $Source_DIR -->debug"
   cmake --build .  --config debug
   echo "************************* $Source_DIR -->release"
   cmake --build .  --config release
   cmake --build .  --target install --config release
   cmake --build .  --target install --config debug
   popd

3. Adjust the CMAKE_GENERATOR_OPTIONS according to your Visual Studio version.

4. In git command line enter the following command:

   ./installOCV.sh

   This script will generate the required installation of Open CV in the current directory (eg. {directory}/AR_Voxel_Project/Libs/install/opencv).

5. Keep in mind that this will take some time.

6. Add {directory}\Libs\install\opencv\bin to PATH.

7. For further information please check the official OpenCV website.

2.3. Other OS

Please refer to the following instructions:

• Eigen: All OS

• Open CV: Linux MacOS

2.4. Build the Project

After installing both libraries, your repository should look like this.

Repository Structure

• Data

  • demo_material

    • images

    • masks

• Libs

  • Eigen3

  • opencv

  • install

    • opencv

  • opencv_contrib

• src

• CMakeLists.txt

• README.adoc

Now you are ready to generate the project using CMake. If you followed the instructions above, CMake should already have found the installation directories of Eigen and OpenCV. Make sure that Eigen3_DIR and OpenCV_DIR are set correctly.

After compiling the project, you are ready to carve your first model.

3. Example

The following example gives visualize the carved model of our demo object. If you want to reproduce our results, you can use this dataset. For comparison we also provide details about the configuration and the expected output meshes.

Figure 1. Example Input Image (total of 8)

Figure 2. Generated Model: post processing and color reconstruction (Method 2 - color averaging) applied.

4. How to run?

4.1. Generate ChArUco Board

Run

$ ./voxel_project.exe -c=1

This command will generate a new subdirectory out containing the file BoardImage.jpg. This ChArUco Board can be used to calibrate your individual camera and later on carve your own models.

4.2. Camera Calibration

Run

$ ./voxel_project.exe -c=2

Perform camera calibration on previously taken images or take images using a connected camera and perform calibration on those.

After initial calibration, the user can interactively choose to exclude specific images by ID (e.g. outliers). For this, reprojection errors are provided for individual images, as well as the overall error.

The final data is saved to a .yml file that can be specified using -calibration

Flag	Default	Description
-live=true/false	true	Whether the images used for calibration should be taken live using a connected camera. If false, a folder with images for calibration has to be provided to -images.
-cam_id=<cam_id>	0	The camera to use when performing live calibration. 0 should be the system’s default device.
-images="<images-dir>"	Ignored if live==true NonOptional if live==false	Directory of the input images to be used for calibration e.g. ./Data/demo_material/images.
-calibration="<calibration-file>"	out/cameracalibration.yml	Output file for the camera calibration data.

4.3. Segmentation

Run

$ ./voxel_project.exe -c=3

TODO

4.4. Carve your own model

Run

$ ./voxel_project.exe -c=5 -images="<images-dir>" -masks="<masks-dir>" -calibration="<cameracalibartion.yml-dir>" -carve=<carving-method> -x=<x-dim> -y=<y-dim> -z=<z-dim> -size=<voxel-size> -scale=<model-scale> -dx=<x-offset> -dy=<y-offset> -dz=<z-offset> -color=<color-method> -model_debug=<model_debug-method> -postprocessing=<postprocessing-method> -intermediateMesh=<intermediateMesh-generation> -outFile=<out_file_path>

This command will generate a new file out/mesh.off containing the mesh generated by carving your specified inputs. To understand more about the flags please refer to the table below.

Table 1. Flags

Flag	Default	Description
-images="<images-dir>"	NonOptional	Directory of the input images e.g. ./Data/demo_material/images.
-masks="<masks-dir>"	NonOptional	Directory of the masks e.g. ./Data/demo_material/masks.
-calibration="<cameracalibartion.yml-dir>"	NonOptional	Directory of the masks e.g. ./Data/demo_material/cameracalibration.yml.
-carve=<carving-method>	1	1 - standard carving 2 - greedy carving (fast)
-x=<x-dim>	100	Number of voxels in x direction.
-y=<y-dim>	100	Number of voxels in y direction.
-z=<z-dim>	100	Number of voxels in z direction.
-size=<voxel-size>	0.0028	Side length of a voxel.
-scale=<model-scale>	1.0	Scale factor for the output model.
-dx=<x-offset>	0.0	Move output model in x direction (unscaled).
-dx=<y-offset>	0.0	Move output model in y direction (unscaled).
-dx=<z-offset>	0.0	Move output model in z direction (unscaled).
-color=<color-method>	0	0 - no color reconstruction 1 - nearest observer 2 - average color
-model_debug=<model_debug-method>	false	true - generate debug model (1 cube ~ 1 visible voxel) false - do not generate debug model
-postprocessing=<postprocessing-method>	true	true - apply postprocessing on the model false - do not apply postprocessing
-intermediateMesh=<intermediateMesh-generation>	false	true - Generates a mesh after each processed image. Only works with carving method 1 false - intermediate mesh generation disabled
-outFile=<out_file_path>	./out/mesh.off	Filepath the generated mesh will be written to. Should end with .off.

4.5. Benchmarking

Run

$ ./voxel_project.exe -c=6 -images="<images-dir>" -masks="<masks-dir>" -calibration="<cameracalibartion.yml-dir>"

This command will execute the provided benchmarking cases on the specified data. For each case a mesh file will be generated in out/bench. Additionally, the bash will print a table containing information about the test cases and execution times (see table below). For more information about the flags please refer to the table above.

Table 2. Benchmark output

Column	Description
Name	Short description of the testcase Model size (Small, Medium, Large) Carving method (1 - standard, 2 - greedy) Coloring method
Model size	Model dimensions (x, y, z direction) and voxel size
Carving time	Time needed to execute carving process (in milliseconds)
Coloring time	Time needed to execute coloring process (in milliseconds)
Postprocessing time	Time needed for postprocessing (in milliseconds)
Marching cubes time	Time needed to transform model into .off-format sutiable representation using the marching cubes algorithm (in milliseconds)
Overall time	Time needed for complete reconstruction process (including output file writing and variable initializations, excluding image loading) (in milliseconds)

TODO

textexample [1] text example [2] te xt exam ple.

References

• [1] Resource 1

• [2] Resource 2