MeshDepth: Disconnected Mesh-based Deep Depth Prediction

Masaya Kaneko 1,2

Ken Sakurada 2

Kiyoharu Aizawa 1

The University of Tokyo 1

National Institute of Advanced Industrial Science and Technology (AIST) 2

  • Input image

  • Depthmap

  • 3D Mesh

Abstract

We propose a novel method for mesh-based single-view depth estimation using Convolutional Neural Networks (CNNs). Conventional CNN-based methods are only suitable for representing simple 3D objects because they estimate the deformation from a predefined simple mesh such as a cube or sphere. As a 3D scene representation, we introduce a disconnected mesh made of 2D mesh adaptively determined on the input image. We made a CNN-based framework to compute depths and normals of faces of the mesh. Because of the representation, our method can handle complex indoor scenes. Using common RGBD datasets, we show that our model achieved best or comparable performance comparing to the state-of-the-art pixel-wise dense methods. It should be noted that our method significantly reduce the number of the parameter representing the 3D structure.

[ Paper ]

[ Code ]

Depthmap Prediction

Quantitative (top) and qualitative (bottom) results showing our depthmap rendered from predicted 3D mesh. Our results achieved the best or comparabe performance to that of the state-of-the-art pixel-wise dense methods (Eigen et al. and Laina et al. ), despite the mesh representation using much less parameters.

Method rel rms log10 delta1 delta2 delta3 #param.
Eigen et al. .158 .641 - .769 .950 .988 921K
Laina et al. .127 .573 .055 .811 .953 .988 921K
Ours .146 .530 .062 .803 .954 .988 32K

Since our depthmap is created by the 2D mesh extracted based on the canny edge, the object boundaries of our depthmap is clearer than that of the pixel-wise method.

Input image GT Ours Eigen et al. Laina et al.

3D Mesh Prediction

These are 3D mesh predictions from a single-view image.
Our CNN framework can be trained end-to-end from general RGBD datasets without GT mesh data.

Video

Citation

@misc{1905.01312,
  Author = {Masaya Kaneko and Ken Sakurada and Kiyoharu Aizawa},
  Title = {MeshDepth: Disconnected Mesh-based Deep Depth Prediction},
  Year = {2019},
  Eprint = {arXiv:1905.01312},
}

Acknowledgements

We thank Shizuma Kubo for modifying the code of this webpage, whose template design was borrowed from keypointnet webpage .