1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
|
.. _optimized_integration:
TSDF Integration
-------------------------------------
Truncated Signed Distance Function (TSDF) integration is the key of dense volumetric scene reconstruction. It receives relatively noisy depth images from RGB-D sensors such as Kinect and RealSense, and integrates depth readings into the :ref:`voxel_block_grid` given known camera poses. TSDF integration reduces noise and generates smooth surfaces.
The integration process mainly consists of two steps, (sparse) **block** selection and activation, and (dense) **voxel** value integration. An example can be found at ``examples/python/t_reconstruction_system/integrate.py``.
Activation
``````````
In the activation step, we first locate blocks that contain points unprojected from the current depth image. In other words, it finds active blocks in the current viewing frustum. Internally, this is achieved by a *frustum* hash map that produces duplication-free block coordinates, and a *block* hash map that activates and query such block coordinates.
.. literalinclude:: ../../../examples/python/t_reconstruction_system/integrate.py
:language: python
:lineno-start: 51
:lines: 8,52-54
Integration
````````````
Now we can process the voxels in the blocks at ``frustum_block_coords``. This is done by projecting all such related voxels to the input images and perform a weighted average, which is a pure geometric process without hash map operations.
We may use optimized functions, along with raw depth images with calibration parameters to activate and perform TSDF integration, optionally with colors:
.. literalinclude:: ../../../examples/python/t_reconstruction_system/integrate.py
:language: python
:lineno-start: 55
:lines: 8,56-63
Currently, to use our optimized function, we assume the below combinations of data types, in the order of ``depth image``, ``color image``, ``tsdf in voxel grid``, ``weight in voxel grid``, ``color in voxel grid`` in CPU
.. literalinclude:: ../../../cpp/open3d/t/geometry/kernel/VoxelBlockGridCPU.cpp
:language: cpp
:lineno-start: 212
:lines: 212-218
and CUDA
.. literalinclude:: ../../../cpp/open3d/t/geometry/kernel/VoxelBlockGridCUDA.cu
:language: cpp
:lineno-start: 238
:lines: 238-244
For more generalized functionalities, you may extend the macros and/or the kernel functions and compile Open3D from scratch to achieve the maximal performance (~100Hz on a GTX 1070), or follow :ref:`customized_integration` and implement a fast prototype (~25Hz).
Surface extraction
``````````````````
You may use the provided APIs to extract surface points.
.. literalinclude:: ../../../examples/python/t_reconstruction_system/integrate.py
:language: python
:lineno-start: 105
:lines: 8, 106-110
Note ``extract_triangle_mesh`` applies marching cubes and generates mesh. ``extract_point_cloud`` uses a similar algorithm, but skips the triangle face generation step.
Save and load
``````````````
The voxel block grids can be saved to and loaded from `.npz` files that are accessible via numpy.
.. literalinclude:: ../../../examples/python/t_reconstruction_system/integrate.py
:language: python
:lineno-start: 47
:lines: 8,48,98
The ``.npz`` file of the aforementioned voxel block grid contains the following entries:
- ``attr_name_tsdf``: stores the value buffer index: 0
- ``attr_name_weight``: stores the value buffer index: 1
- ``attr_name_color``: stores the value buffer index: 2
- ``value_000``: the tsdf value buffer
- ``value_001``: the weight value buffer
- ``value_002``: the color value buffer
- ``key``: all the active keys
- ``block_resolution``: 8
- ``voxel_size``: 0.0059 = 3.0 / 512
- ``CUDA:0``: the device
|
