Cheng Sun

About me

I am a research scientist at NVIDIA Research Taiwan, specializing in 3D reconstruction, understanding, and their practical applications. I earned my Ph.D. from the Institute of Information Systems and Applications at National Tsing Hua University. Throughout my doctoral studies, I had the privilege of collaborating with Prof. Hwann-Tzong Chen at VLLab and Prof. Min Sun at VSLab.

Find my resume here (last update 05/21/2023).

Timeline

2023-09 NVIDIA research scientist, Taipei
2023-08 BMVC 2023 ✨ 360 Data Augmentation
2023-07 ICCV 2023 ⭐️✨✨ Physically-based Inverse Rendering, Video Editing, Point Cloud Object Detection
2022-07 ECCV 2022 ✨✨ 3D Morphing, Point Cloud Semantic Segmentation, 3D Model Pre-training
2022-06 Meta research scientist intern, Redmond
2022-05 Google conference scholarship
2022-03 CVPR 2022 ⭐️ Grid-based NeRF, Speed-up NeRF Optimization
2021-07 ICCV 2021 ⭐️ Semantic Segmentation
2021-05 MICCAI 2021 ✨ Medical Imaging, Weakly-supervised 3D Object Detection
2021-03 CVPR 2021 ⭐️⭐️ 360 Depth Estimation, 360 Room Layout, 360 Semantic Segmentation, 360 Planes Detection
2020-09 ASUS AICS PhD fellowship program
2020-08 ECCV 2020 Holistic 3D Vision Challenges 1st place
2019-12 WACV 2020 ✨ 360 Object Detection
2019-07 Novatek Graduate Student Scholarship
2019-03 CVPR 2019 ⭐️ 360 Room Layout
2018-06 IROS 2018 ✨ HCI, Object Detection
⭐️: A first or co-first author paper / ✨: A co-author paper

Publications

Neural-PBIR Reconstruction of Shape, Material, and Illumination

Cheng Sun*, Guangyan Cai*, Zhengqin Li, Kai Yan, Cheng Zhang, Carl Marshall, Jia-Bin Huang, Shuang Zhao, Zhao Dong

ICCV 2023

Object Shape, Physically-based Material, Lighting

[Paper] [Project Page]

Hashing Neural Video Decomposition with Multiplicative Residuals in Space-Time

Cheng-Hung Chan, Cheng-Yang Yuan, Cheng Sun, Hwann-Tzong Chen

ICCV 2023

Video Editing, Test-time Training

[Project Page]

ImGeoNet: Image-induced Geometry-aware Voxel Representation for Multi-view 3D Object Detection

Tao Tu, Shun-Po Chuang, Yu-Lun Liu, Cheng Sun, Ke Zhang, Donna Roy, Cheng-Hao Kuo, Min Sun

ICCV 2023

Point Cloud, Object Detection, 2D-3D Fusion

[Project Page]

Multiview Regenerative Morphing with Dual Flows

Chih-Jung Tsai, Cheng Sun, Hwann-Tzong Chen

ECCV 2022

3D Morphing, Grid-based NeRF

[Paper] [Codes] [Project Page]

Data Efficient 3D Learner via Knowledge Transferred from 2D Model

Ping-Chung Yu, Cheng Sun, Min Sun

ECCV 2022

Point Cloud, Semantic Segmentation, Pre-training, Semi-supervised Learning

[Paper] [Codes]

Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction

Cheng Sun, Min Sun, Hwann-Tzong Chen

CVPR 2022 (oral)

Grid-based NeRF

[Paper] [Paper v2] [Codes] [Project Page] [Video]

It's one of the first method to speedup NeRF optimization using grid-based models. NeRF takes 20+ hours to optimize a scene while our v2 takes only 5 minutes with comparable quality.

Specialize and Fuse: Pyramid Output Representation for Semantic Segmentation

Chi-Wei Hsiao*, Cheng Sun*, Hwann-Tzong Chen, Min Sun

ICCV 2021

Semantic Segmentation

[Paper]

HoHoNet: 360 Indoor Holistic Understanding with Latent Horizontal Features

Cheng Sun, Min Sun, Hwann-Tzong Chen

CVPR 2021

360 Panorama, Semantic Segmentation, Depth, Room layout

[Paper] [Codes] [Project Page] [Video]

Indoor Panorama Planar 3D Reconstruction via Divide and Conquer

Cheng Sun, Chi-Wei Hsiao, Ning-Hsu Wang, Min Sun, Hwann-Tzong Chen

CVPR 2021 (oral)

360 Panorama, Planes Detection, New Dataset

[Paper] [Codes] [Video]

Leveraging Auxiliary Information from EMR for Weakly Supervised Pulmonary Nodule Detection

Hao-Hsiang Yang, Fu-En Wang, Cheng Sun, Kuan-Chih Huang, Hung-Wei Chen, Yi Chen, Hung-Chih Chen, Chun-Yu Liao, Shih-Hsuan Kao, Yu-Chiang Frank Wang, Chou-Jin Lan

MICCAI 2021

MRI, 3D Detection, Weakly-supervised Learning

[Paper]

360-Indoor: Towards Learning Real-World Objects in 360° Indoor Equirectangular Images

Shih-Han Chou, Cheng Sun, Wen-Yen Chang, Wan-Ting Hsu, Min Sun, Jianlong Fu

WACV 2020

360 Panorama, Object Detection, New Dataset

[Paper] [Dataset] [Project Page]

Flat2Layout: Flat Representation for Estimating Layout of General Room Types

Chi-Wei Hsiao, Cheng Sun, Min Sun, Hwann-Tzong Chen

Room Layout

[Paper]

HorizonNet: Learning Room Layout with 1D Representation and Pano Stretch Data Augmentation

Cheng Sun, Chi-Wei Hsiao, Min Sun, Hwann-Tzong Chen

CVPR 2019

360 Panorama, Room layout, Data Augmentation

[Paper] [Codes] [Project Page]

We introduced a novel 360 room layout parameterization, which is now widely adopted in this task. It won the 1st place in ECCV 2020 Holistic 3D Vision Challenge. It was also adopted as the strong baseline in Kujiale's Structure3D and Zillow's ZInD official benchmarks.

DLWV2: a Deep Learning-based Wearable Vision-system with Vibrotactile-feedback for Visually Impaired People to Reach Objects

Meng-Li Shih, Yi-Chun Chen, Chia-Yu Tung, Cheng Sun, Ching-Ju Cheng, Liwei Chan, Srenivas Varadarajan, Min Sun

IROS 2018

Object Detection, HCI

[Paper]

Education

PhD, EECS, National Tsing Hua University

Sep 2018 - Aug 2023

GPA – 4.28 / 4.3

Advisors: Prof. Hwann-Tzong Chen (at VLLab) and Prof. Min Sun (at VSLab).

I transformed it from an MS program to a Ph.D. program in Aug 2020.

BS, CS, National Chiao Tung University

Sep 2014 - Feb 2018

GPA – 3.87 / 4.0

It is now called National Yang Ming Chiao Tung University.

Experience

Research Scientist, Nvidia Research Taiwan team

Sep 2023 - Now

Phi Tau Phi Scholastic Honor Society Honorary Membership

Apr 2023

Awarded to students with outstanding academic performance.

Computer Vision Research Intern, Aeolus Robotics

Feb 2023 - Aug 2023

Indoor 3D reconstruction from robot sensor / Generic 3D semantic and instance segmentation

Research Scientist Intern, Meta Reality Labs

June 2022 - Dec 2022

Object 3D reconstruction from RGB video / Physically-based material estimation / Lighting estimation

Google Conference Scholarship

May 2022

Thanks google for my trip from Taiwan to New Orleans.

Ph.D. Scholarship, ASUS Intelligent Cloud Services (AICS)

Sep 2020 - Apr 2022

Awarded to phd students with outstanding research ability.

Research Scientist Intern, Appier

Dec 2019 - Feb 2020

Tabular data / Clickthrough rate estimation.

M.S. Scholarship, Novatek

Aug 2019 - July 2020

Awarded to master students with outstanding research ability.