Name: 3D-COCO
Creator: Maxence Bideaux
License: https://creativecommons.org/licenses/by/4.0/
Keywords: Computer Vision

Abstract

3D-COCO is a dataset composed of MS COCO images with 3D models aligned on each instance. 3D-COCO was designed to achieve computer vision tasks such as 3D reconstruction or image detection configurable with textual, 2D image, and 3D CAD model queries.

3D-COCO is an extension of the original MS-COCO dataset providing 3D models and 2D-3D alignment annotations. We complete the existing MS-COCO dataset with 28K 3D models collected on ShapeNet and Objaverse. By using an IoU-based method, we match each MS-COCO annotation with the best 3D models to provide a 2D-3D alignment.

The zip file contains the original COCO dataset, the 3D models, the enriched annotations files with the models as well as the code to obtain such annotations with any image and 3D CAD

Instructions:

# 3D-COCO : Readme

Requirements :
- A huggingface account is needed
- A conda installation is needed

## 1. Create environment

There are two conda environnements, using one for rendering

```
conda env create --file=scripts/COCO3D_dependencies.yaml
conda activate COCO3D_collection
```

## 2. Download the datasets

Define your root path to put the dataset folder

```
cd /path/to/root
mv scripts ./

mkdir COCO3D
mkdir COCO3D/Annotations
mv ./scripts/COCO3D_annotation_train2017.json ./scripts/COCO3D_annotation_val2017.json ./COCO3D/Annotations/
```

Download Shapenet from https://huggingface.co/datasets/ShapeNet/ShapeNetCore-archive/blob/main/...

```
unzip ShapeNetCore.v2.zip -d ./COCO3D/
mv ./COCO3D/ShapeNetCore.v2 ./COCO3D/ShapeNet
mkdir ./COCO3D/Models
python3 scripts/select_shapenet_models.py ./COCO3D
```

Download Objaverse
```
bash scripts/download_objaverse.sh
```

Download COCO Images
```
bash scripts/download_coco.sh
```

## 3.1 Renders Synthetics Images

Render untextured Images and Depth Maps using Blender 2.77
```
bash scripts/install_rendering.sh
```

```
conda activate renderenv
python3 ./scripts/create_renders.py ./COCO3D
```

## 3.2 Convert to point cloud or voxel
```
conda activate COCO3D_collection
python3 scripts/convert_to_voxel.py ./COCO3D
python3 scripts/convert_to_pointcloud.py ./COCO3D
```

Funding Agency:

European Union

Grant Number:

101073951